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USING DIGITAL OBJECTS ORGANIZED 
ACCORDING TO A HISTOGRAM TIMELINE 
CROSS REFERENCE TO RELATED APPLICATIONS 

Reference is made to commonly assigned U.S. Patent Application 
5 Serial No. 09/742,028 filed December 20, 2000, entitled "Timeline-Based 

Graphical User Interface for Efficient Image Database Browsing and Retrieval" 
by Elizabeth Rosenzweig et al; commonly assigned U.S. Patent Application 
Serial No. 09/163,618 filed September 30, 1998, entitled "A Method for 
Automatically Classifying Images Into Events" by Alexander C. Loui et al.; and 
10 commonly assigned U.S. Patent Application Serial No. 09/197,363 filed 

November 20, 1998, entitled "A Method for Automatically Comparing Content of 
Images for Classification Into Events" by Alexander C. Loui et al., the disclosures 
of which are incorporated herein by reference. 

FIELD OF THE INVENTION 
15 The present invention relates to an effective method of organizing 

digital objects and for facilitating the display or processing of desired objects. 
BACKGROUND OF THE INVENTION 
Digital imaging devices are being used with increasing frequency 
to record digital still images and digital audio/video sequences. Digital multimedia 
20 content from other sources is also proliferating, such as digitized consumer 

snapshots, digitized home video, digitally scanned graphics, digitized music, and 
other audio recording. With the potential for vast numbers of such multimedia 
objects to be generated and stored by individuals, the problem of representing a 
collection of such objects to a user for retrieval and organization becomes 
25 important. Typically, a digital camera user takes a plurality of digital images or 
audio/video sequences that are stored on a removable memory card. These 
multimedia objects can be transferred from the memory card and stored, for 
example, on a hard drive or other non-volatile memory associated with the user's 
computer. Alternatively, digital multimedia objects can be transferred over the 
30 internet to a web site for storage, sharing, printing, and other purposes. 

Over time, a user can collect hundreds, or perhaps thousands, of 
multimedia objects, which are records of events at various points in time. As the 
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number of such objects in a particular database grows, traditional methods of 
displaying them to a user such as in hierarchical file structures or simple "contact 
sheets" of image thumbnails become ineffective. 

SUMMARY OF THE INVENTION 
5 It is an object of the present invention to permit a user to facilitate 

access to desired digital objects from a database. 

It is another object of the present invention to provide an effective 
method for organizing representations of digital multimedia objects to facilitate 
the selection of desired digital multimedia objects. 
10 These objects are achieved by a method for organizing visual 

digital objects and for selecting one or more of such visual digital objects for 
viewing, comprising the steps of: 

a) developing a histogram timeline which identifies a number 
of visual digital objects organized according to predetermined time periods and 

15 providing thumbnail representations thereof; 

b) selecting a portion of the histogram timeline for viewing 
such thumbnail representations of visual digital objects corresponding to such 
selected portion; and 

c) determining if one or more of the viewed such thumbnail 
20 representations is of interest and then viewing the corresponding digital visual 

object(s). 

These digital objects can be digital still images, digital audio files, 
digital video segments, graphics files, or related multimedia data. It is an 
important feature of the present invention that the use of a histogram timeline 
25 permits effective interaction with selecting either single media or multimedia 
digital objects in a database. 

ADVANTAGES 

It is an advantage of the present invention that the generation of a 
histogram timeline facilitates the representation of a collection of multimedia 
30 objects to a user and to facilitate the interaction and selection of the objects. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram of a system for practicing the present 

invention; 

FIG. 2 is a histogram timeline diagram made in accordance with 
5 the present invention; 

FIG. 3A is a flow diagram showing how a histogram timeline, such 
as in FIG. 2, can be made and further describing how to select a portion of the 
histogram timeline for viewing such thumbnail representations of such objects 
corresponding to such selected portion; 
10 FIG. 3B is a flow diagram showing the determination of the desired 

span of the histogram timeline and showing how to select a portion of the 
histogram timeline for viewing such thumbnail representations of such objects 
corresponding to such selected portion; 

FIG. 4 is a flow diagram showing, in more detail, how a histogram 
1 5 timeline can be made; 

FIG. 5 is a flow diagram showing how the histogram timeline can 
be made responsive to the resolution and size of the display device; 

FIG. 6A is a depiction of a histogram timeline on a display screen 
in an initial state; 

20 FIG. 6B is a depiction of a histogram timeline and thumbnail 

representations on a display screen after a portion of the timeline has been 
selected; 

FIG. 6C is a depiction of a histogram timeline and thumbnail 
representations on a display screen during selection of a shortened time scale; 
25 FIG. 6D is a depiction of a histogram timeline without thumbnail 

representations on a display screen during selection of a shortened time scale; 

FIG. 6E is a depiction of a histogram timeline on a display screen 
after selection of a shortened time scale; 

FIG. 6F is a depiction of a histogram timeline and thumbnail 
30 representations on a display screen during further selection of a shortened time 
scale; and 
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FIG. 6G is a depiction of a histogram timeline and thumbnail 
representations on a display screen during further selection of a shortened time 
scale. 

DETAILED DESCRIPTION OF THE INVENTION 

The present invention provides an effective method of representing 
a collection of multimedia or single media objects to a user and to facilitating the 
interaction with and selection of such objects. These objects are typically visual 
digital objects such as, for example, still image files, MPEG-1 motion sequences 
with sound, or digital audio objects such as "WAV" files, that can be collected by 
a user and distributed over a variety of media and storage locations. 

The entire collection of these objects, or even some subset, can be 
so large as to make searching, annotating, processing, or viewing difficult. 

With the notable exception of recorded music, the majority of these 
objects are, to the user, strongly associated with a date or event. This date is 
typically, though not necessarily, the date of the origination of the object data, as 
is the case with personal memories such as still images, sound recordings, and 
home video sequences. 

Since each of these objects is typically associated with a date, it is 
natural to organize them according to dates. A timeline is a well known method 
for showing events in chronological order, and representations of objects, such as 
thumbnail images, can be arranged on a timeline for display. However, in large 
collections more than one object can be associated with a particular date, and the 
collection can cover a long time span. In such cases, it is impractical to arrange 
thumbnails of all objects along the timeline. In accordance with the present 
invention, a more effective representation is to build a histogram timeline in 
which the number of objects associated with each date, or time interval, are 
represented in a graphical form such as a bar chart. More particularly, as will be 
seen, the histogram timeline represents visual digital objects organized according 
to predetermined time periods. Thereafter, thumbnail representations are produced 
after a selection from the histogram timeline. 

As shown in FIG. 1, a system for practicing the present invention 
includes a home computer 10, which can be, for example, a Dell Dimension L. 
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The home computer 10 includes a CPU 14, which communicates with other 
devices over a bus 12. The CPU 14 executes software stored on a hard disk drive 
20, for example, the well known Windows 98 operating system software provided 
by Microsoft Corp. of Redmond, WA. A video display device 52 is coupled to the 

5 CPU 14 via a display interface device 24. The mouse 44 and keyboard 46 are 
coupled to the CPU 14 via a desktop interface device 28. The home computer 10 
also contains a CD-R/W drive 30 to read various CD media and write to CD-R or 
CD-RW writable media 42. A floppy disk drive 32 is also included to read from 
and write to floppy disks 40. An audio interface device 26 connected to bus 12 

10 permits audio data from, for example, a digital sound file stored on hard disk drive 
20, to be converted to analog audio signals suitable for speaker 50. The audio 
interface device 26 also converts analog audio signals from microphone 48 into 
digital data suitable for storage in, for example, the hard disk drive 20. In addition, 
the home computer 10 is connected to an external network 60 via a network 

15 connection device 1 8. A digital camera 6 can be connected to the home computer 
10 through, for example, the USB interface device 34 to transfer still images, 
audio/video, and sound files from the camera to the hard disk drive 20 and vice- 
versa. 

A collection of digital multimedia or single-media objects (digital 
20 images) can reside exclusively on the hard disk drive 20, compact disk 42, or at a 
remote storage device such as a web server accessible via the network 60. The 
collection can be distributed across any or all of these as well. 

To represent the collection, first all of the digital multimedia 
objects must be associated together as a database. It will be understood that these 
25 digital multimedia objects can be digital still images, such as those produced by 
digital cameras; audio data, such as digitized music or voice annotation files in the 
"WAV" or "MP3" audio file format; and digital video segments with or without 
sound, such as MPEG-1 or MPEG-4 video. Digital multimedia objects also 
include files produced by graphic software, for example the well known Visio 
30 graphics software product, and files representing text documents, such as those 
produced by the well known Microsoft Word product. A database of digital 
multimedia objects can comprise only one type of object or any combination. 
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Once a collection of digital multimedia objects are associated 
together to form a database, they can be represented to the user in accordance with 
the present invention. FIG. 2 shows an example of a histogram timeline. In this 
example, the horizontal axis is enumerated in units of time and the vertical axis is 
5 enumerated in the number of digital multimedia objects. Alternatively, the vertical 
axis can be enumerated in the relative number of digital multimedia objects. The 
length of graphical bar 80 represents the relative number of objects in the chosen 
database in a given date bin on timelines 82 and 84. A date bin is the interval of 
time over which objects associated with a date in that interval are counted 

10 together. A date bin can have any duration, but would typically be no less than 
one day. Larger date bins might be constructed if the display area for the 
histogram was small or the length of the timeline was so large that short date bins 
could not be displayed adequately. In this example, time scale 82 is composed of 
month increments and time scale 84 is composed of year increments, though a 

15 single time scale is also possible. 

FIG. 3 A is a flow diagram that illustrates the generation of the 
histogram timeline and the interactive steps for practicing the present invention. In 
step 100, the histogram timeline is formed by counting the number of objects in 
the database that are associated with each date. The result, internally, is an array 

20 of ordered pairs of numbers. One number in the pair represents a given date, and 
the associated number represents the number of objects associated with the date. 

In step 102, the histogram timeline is displayed graphically to the 
user, as shown in FIG. 6 A, with a predetermined range of dates (timeline) based, 
for example, on the earliest to latest dates associated with all of the objects in the 

25 chosen database. Region 600 denotes the area in which the timeline is located, and 
604 and 606 are the fine and coarse timeline scales, respectively. Region 602 is a 
separate viewable area where iconic or thumbnail representations of digital 
multimedia objects can be displayed to the user. The user may also select one or 
more of the representations to view, listen to, and/or process. Pointer 610 is a 

30 selection device such as a mouse, joystick, or eye movement tracking apparatus. 

Referring again to FIG. 3 A, in step 104 the user selects a date of 
interest on the histogram timeline. This is depicted in FIG. 6B, where the user 
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selects a date by moving the mouse to move a graphical pointer 610 to a particular 
histogram bar on the timeline and "clicking" the mouse button. If the date bins 
were longer than one day, then "clicking" on a histogram bar would select the 
range of dates included in the date bin interval. After selection of the date, 
5 thumbnail or iconic representations appear, corresponding to the objects 

associated with the chosen date, according to step 106 in FIG. 3 A. Additionally, 
the thumbnail and iconic representations displayed, can be grouped according to 
the events to which they relate, as described in commonly assigned U.S. Patent 
Application Serial No. 09/163,618 filed September 30, 1998, entitled "A Method 

10 for Automatically Classifying Images Into Events" by Alexander C. Loui et al., 
and commonly assigned U.S. Patent Application Serial No. 09/197,363 filed 
November 20, 1998, entitled "A Method for Automatically Comparing Content of 
Images for Classification Into Events" by Alexander C. Loui et al., the disclosures 
of which are incorporated herein by reference. 

15 In step 108, the user decides if the representations displayed are of 

interest. If not, the user can then select a new date on the histogram timeline, as 
shown in step 1 10. In step 112, new iconic or thumbnail representations are 
displayed corresponding to objects associated with the new date. Alternatively, 
other ways can be used to access thumbnail or iconic representations extending 

20 just outside of the range displayed to the user. For example, a scroll bar oriented 
below the representations could be used to scroll forward and backward in time, 
with the representations and the timeline changing in response. In the practice of 
the present invention, it will be understood that "dragging" a thumbnail to a new 
viewing position can be accomplished by selecting it and then moving the pointer. 

25 As the thumbnail is dragged, thumbnail representations in the direction of motion 
are "pushed" off the screen, and new thumbnail representations are "pulled" into 
the screen from the opposite side. 

Referring again to FIG. 3 A, if one or more of the thumbnail 
representations is of interest, the user can then move on to view or process one of 

30 the objects represented as shown in step 114. Processing can entail, for example, 
invoking an image viewer for viewing an image at a higher resolution, invoking 
an image editor to edit a still image, invoking an audio player for listening to 



audio data, or invoking a video player to view a video sequence. Processing can 
also entail the addition of audio, graphical, and text annotation to an object, 
transferring objects to another database or collection, or some other function based 
on the digital multimedia object type and user's desire. 
5 FIG. 3B is a flow diagram that illustrates the generation of a 

histogram timeline and further steps involved in interacting with it. The histogram 
timeline is formed as shown in step 130 by counting the number of objects in the 
database that are associated with each date. The result, internally, is an array of 
ordered pairs. One number in the pair represents a given date, and the associated 

10 number represents the number of objects associated with the date. 

In step 132, the histogram timeline is displayed graphically to the 
user, as shown in FIG. 6A, with a predetermined range of dates (timeline) based, 
for example, on the earliest to latest dates associated with all of the objects in the 
chosen database. The histogram is displayed with two time scales; a primary time 

15 scale having coarse intervals, such as whole years, and an adjacent secondary time 
scale having finer intervals such as whole months. 

In step 134, the user determines whether or not the timeline span is 
too large to work with effectively. If so, then in step 136 the user selects an 
interval either on the coarse primary time scale or the finer secondary time scale. 

20 FIG. 6C illustrates selection of the year 1999 on the coarse time scale, with 

representations of randomly selected objects from that year appearing while the 
date is selected by, for example, holding down a mouse button. In addition, the 
region denoting the year 1999 is highlighted to suggest selection. The appearance 
of the representations can persist as long the mouse button is held down by the 

25 user. Alternatively, if the display of representations is slow relative to the amount 
of time the user wishes to hold down the mouse button, only the region denoting 
the year 1999 is highlighted without displaying any representations, as illustrated 
in FIG. 6D. 

Upon release of the mouse button by the user, the histogram 
30 timeline is redisplayed according step 138 of FIG. 3B, with the earliest and latest 
dates of the histogram timeline changed to correspond with span of the interval 
selected, as illustrated in FIG. 6E. If the process is repeated, then the primary and 



secondary timeline intervals are changed as appropriate. For example, if the user 
selects the month of May, 1999 as shown in FIG. 6F, then new thumbnail or 
iconic representations corresponding to objects associated with May, 1999 are 
displayed while the mouse button is held down by the user. When the user 
5 releases the mouse button, histogram timeline is redisplayed such that the earliest 
and latest dates of the histogram timeline correspond with the month of May, 
1999. The coarse intervals are then months and the fine intervals days as 
illustrated in FIG. 6G. Selection of any day in the month, also shown in FIG. 6G, 
results in thumbnail or iconic representations to be displayed, corresponding to 

10 digital multimedia objects associated with that day. 

At any point in the process, as shown in step 144 of FIG. 3B, the 
user can select a date by selecting a bar on any of the histogram timeline 
representations, as illustrated earlier in FIG 6B. Representations of digital 
multimedia objects corresponding to the date are then displayed as shown in step 

15 146. If the user was interested in a particular thumbnail representation according 
to step 142, the user could then view or otherwise interact with the actual object or 
set of objects by, for example, "double clicking" on a representation, as shown in 
step 148. 

FIG. 4 is a flow diagram that describes in more detail how a 
20 histogram timeline in accordance with the present invention can be generated. In 
step 210, each multimedia object in the database is accessed and a date of 
origination is established from either header data, such as capture date from a 
digital camera, or from some other data associated with the object. For example, 
in the absence of reliable header information, the object's file origination date 
25 could be used. Alternatively, user-entered annotation could be searched. In step 
212, the number of objects associated with each date are determined, to form a 
histogram timeline. Finally, in step 214, the histogram timeline is displayed to the 
user. 

FIG. 5 depicts a flow diagram showing how the histogram timeline 
30 can be made responsive to the resolution and size of the display device. This can 
be important if the histogram is to be displayed on a portable device such as a 
cellular phone or personal digital assistant where display resolution and size are 
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limited. It is also useful on any display if the time span of the histogram timeline 
is very long and individual date bins cannot be adequately displayed. 

Once the histogram timeline is generated, the start and end dates of 
the histogram timeline to be displayed are determined as shown in step 230. In 
5 step 232, the resolution and physical size of the display area are determined. For 
example, resolution could be expressed as the number of horizontal and vertical 
pixels constituting the display area, and the size of the display area could be 
expressed as horizontal and vertical dimensions in millimeters. 

Next, in step 234, the number of histogram bins that can be clearly 
10 displayed is calculated. The number of histogram bins that can be clearly 

displayed can, for example, include additional factors such as a human visual 
system model. 

In step 236, based on the number of histogram bins that can be 
clearly displayed and the desired start and end date of the histogram timeline, new 

15 date bin sizes are calculated. In step 238, a new histogram timeline is generated 
with the modified bin sizes. Finally, in step 240, the modified histogram timeline 
with more visible bin size is displayed. 

A computer program product can include one or more storage 
medium, for example; magnetic storage media such as magnetic disk (such as a 

20 floppy disk) or magnetic tape; optical storage media such as optical disk, optical 
tape, or machine-readable bar code; solid-state electronic storage devices such as 
random access memory (RAM), or read-only memory (ROM); or any other 
physical device or media employed to store a computer program having 
instructions for practicing a method according to the present invention. 

25 The invention has been described in detail with particular reference 

to certain preferred embodiments thereof, but it will be understood that variations 
and modifications can be effected within the spirit and scope of the invention. 
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PARTS LIST 



6 


digital camera 


10 


home computer 


12 


data bus 


14 


CPU 


16 


read-only memory 


18 


network connection device 


20 


hard disk drive 


22 


random access memory 


24 


display interface device 


26 


audio interface device 


28 


desktop interface device 


30 


CD-R/W drive 


32 


floppy disk drive 


34 


USB interface device 


40 


floppy disk 


42 


CD-RW writable media 


44 


mouse 


46 


keyboard 


48 


microphone 


50 


speaker 


52 


video display device 


60 


network 


80 


graphical bar 


82 


timeline scale 


84 


timeline scale 


100 


block 


102 


block 


104 


block 


106 


block 


108 


block 
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PARTS LIST (can't) 



110 


block 


112 


block 


114 


block 


130 


block 


132 


block 


134 


block 


136 


block 


138 


block 


142 


block 


144 


block 


146 


block 


148 


block 


210 


block 


212 


block 


214 


block 


230 


block 


232 


block 


234 


block 


236 


block 


238 


block 


240 


block 


600 


timeline location 


602 


viewable area 


604 


timeline scale 


606 


timeline scale 


610 


pointer 



